Automated network device provisioning

ABSTRACT

A device provisioning configuration request from a network device to be configured is received. The network device provides the request to a network location obtained via a Dynamic Host Configuration Protocol request made automatically by the network device. A configuration agent script is provided in response to the device provisioning configuration request. One or more notifications associated with a provisioning progress are received from the network device via the configuration agent script. At least one of the one or more notifications is utilized in a provisioning workflow.

BACKGROUND OF THE INVENTION

Traditional network provisioning techniques require significant manualintervention and oversight. Each network device requires generating aconfiguration plan that may include running thirty or more differentprovisioning jobs and may utilize dozens of different provisioningtools. Once a configuration plan is generated, a human operator mustensure that each of the steps for provisioning is successfullycompleted. A failure in any step can require a significant timeinvestment that involves manually diagnosing the failure and retryingfailed provisioning steps. Moreover, the amount of effort involvedincreases significantly when a group of devices must be provisionedtogether. The number of steps and tools required increases quickly andthe complexity rises. A provisioning mistake for any one device cancause the device as well as others on the network to operateinefficiently, become inoperable, or expose security flaws. Althoughcertain configuration frameworks such as Zero Touch Provisioning (ZTP)can automate portions of the provisioning process, ZTP techniques arelimited to running jobs from the network device being provisioned andprovide limited to no feedback on the status of those jobs. Therefore,there exists a need for an automated provisioning workflow that allowsone or more network devices to be provisioned including runningprovisioning jobs related to both the network device and the networkinfrastructure. The workflow should be capable of managing a variety ofdifferent configurations and minimizes the need for human interventionwhile allowing the process to be monitored remotely.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the followingdetailed description and the accompanying drawings.

FIG. 1 is a block diagram illustrating an embodiment of a networkinfrastructure for automating network device provisioning.

FIG. 2 is a functional diagram illustrating a programmed computer systemfor automating network device provisioning.

FIG. 3 is a flow diagram illustrating an embodiment of a process forautomating network device provisioning.

FIG. 4 is a flow diagram illustrating an embodiment of a process for aprovisioning server for automating network device provisioning.

FIG. 5 is a flow diagram illustrating an embodiment of a process forautomating network device provisioning.

FIG. 6 is a flow diagram illustrating an embodiment of a process forreceiving feedback from a provisioning agent for a network device.

DETAILED DESCRIPTION

The invention can be implemented in numerous ways, including as aprocess; an apparatus; a system; a composition of matter; a computerprogram product embodied on a computer readable storage medium; and/or aprocessor, such as a processor configured to execute instructions storedon and/or provided by a memory coupled to the processor. In thisspecification, these implementations, or any other form that theinvention may take, may be referred to as techniques. In general, theorder of the steps of disclosed processes may be altered within thescope of the invention. Unless stated otherwise, a component such as aprocessor or a memory described as being configured to perform a taskmay be implemented as a general component that is temporarily configuredto perform the task at a given time or a specific component that ismanufactured to perform the task. As used herein, the term ‘processor’refers to one or more devices, circuits, and/or processing coresconfigured to process data, such as computer program instructions.

A detailed description of one or more embodiments of the invention isprovided below along with accompanying figures that illustrate theprinciples of the invention. The invention is described in connectionwith such embodiments, but the invention is not limited to anyembodiment. The scope of the invention is limited only by the claims andthe invention encompasses numerous alternatives, modifications andequivalents. Numerous specific details are set forth in the followingdescription in order to provide a thorough understanding of theinvention. These details are provided for the purpose of example and theinvention may be practiced according to the claims without some or allof these specific details. For the purpose of clarity, technicalmaterial that is known in the technical fields related to the inventionhas not been described in detail so that the invention is notunnecessarily obscured.

The automation of network device provisioning using a provisioning agentis disclosed. By issuing a single device provisioning command, one ormore network devices are each automatically provisioned using aprovisioning agent provided to each device and a generated configurationplan based on that device's attributes. For example, a network device,such as a switch or router, is installed into a network data center. Aprovisioning command is initiated via a provisioning server. Aconfiguration plan for the device is generated and all requiredprovisioning jobs of the plan, including jobs external to the networkdevice, as well as jobs local to the network device, are completed. Jobslocal to the network device are performed by providing an agent to beinstalled on the device and running provisioning tasks via the agent.Prior to installing the agent, pre-agent provisioning jobs areperformed. Examples of pre-agent provisioning jobs may include settingup BGP peers, draining traffic, inventory asset management tasks, and/orassigning an IP address, among others. Once pre-agent jobs arecompleted, the agent is provided to the targeted network device andprovisioning jobs local to the device are performed via the agent.Examples of local jobs include installing software, configuring softwareon the device, setting up local firewall rules, and/or determiningnetwork connectivity from the perspective of the device, among others.In some embodiments, agent provisioning utilizes and conforms to a ZeroTouch Provisioning (ZTP) framework supported by certain hardware devicemanufacturers for configuring network equipment. While performingagent-initiated provisioning jobs, the agent can update the provisioningserver with feedback, such as the status of agent provisioning jobs. Forexample, the agent can send feedback corresponding to the start of agentprovisioning, status and/or completion of one or more jobs related toagent provisioning, and feedback indicating the completion of agentprovisioning. Once agent provisioning is complete, post-agentprovisioning jobs are performed. For example, during post-agentprovisioning, network connectivity/protocols can be validated, trafficcan be undrained, and/or notifications related to device status can besent, among others. In various embodiments, one or more provisioningjobs can be run simultaneously or in parallel. In some configurations,one or more devices can be provisioned together. For example, a group orplane of network devices can be provisioned together in parallel orsequentially, as appropriate.

By utilizing the disclosed techniques, one or more network devices canbe configured with ease. By automating device provisioning, the amountof human monitoring required for installing, configuring,troubleshooting, and/or updating network devices is greatly reduced. Forexample, when a provisioning task for a device fails, a humanadministrator does not need to physically access the network deviceand/or log into the device to determine which jobs of the configurationplan were successful, which jobs failed, the cause of the failure(s),and/or how to restart or complete the provisioning process. Moreover,pre-agent, agent, and post-agent jobs can be performed in their properintended sequence and dependencies on the job orders can be strictlyenforced. For example, some jobs may first require that the device beassigned an IP address, while others may require first knowing the portthe device is plugged into. The disclosed automated provisioningtechniques allow for the strict enforcement of these types ofdependencies. And by applying one or more of these techniques acrossmultiple devices, a human administrator can more easily install andupdate tens, hundreds, or more network devices at once, each withpotentially different configuration plans. For example, a group ofdevices can be easily refreshed when a new configuration is available.As another example, each week an administrator can refresh theconfiguration of the entire network (or a subset or plane of thenetwork) with a single provisioning command for a clean installation. Asyet another example, experimental configurations can be pushed out to asubset of devices for testing and then the subset of devices can laterbe reverted to a stable configuration when testing is completed. Theability to automate these provisioning jobs greatly improves theefficiency of network administration and consequently improves thefunctional stability and performance of the overall network. Moreover,via the provisioning server, the provisioning status of each device canbe monitored during this process and a global view of the status ofprovisioning is provided. These global monitoring features help ensurethat each device is properly provisioned and simplify troubleshootingwhen failures do occur.

In some embodiments, a device provisioning configuration request isreceived from a network device to be configured. The network deviceprovides the request to a network location obtained via a Dynamic HostConfiguration Protocol (DHCP) request made automatically by the networkdevice. For example, a network device is physically installed in a datacenter and provided power and physical network connectivity. Oncepowered on, as part of a Zero Touch Provisioning (ZTP) configurationprocess, the network device automatically makes a DHCP request. The DHCPrequest includes an option field and device information (vendor and/ormake, etc.) identifying the DHCP request as a ZTP-compatible DHCPrequest. In response to the DHCP request, a network location for sendinga provisioning configuration request is provided. After receiving thenetwork location for a provisioning configuration request, the networkdevice makes a provisioning configuration request to the correspondingprovisioning server. In some embodiments, a configuration agent scriptis provided in response to the device provisioning configurationrequest. For example, the provisioning server provides a provisioningagent in the body of an HTTP-REPLY to a provisioning configurationHTTP-GET request. Once the agent is successfully downloaded, the networkdevice executes the provisioning agent to perform provisioning jobs fromthe network device. In some embodiments, a notification is receivedassociated with the provisioning progress from the network device viathe configuration agent. For example, the provision agent providesprogress feedback notification(s) to the provisioning server.Notifications may include start and complete feedback notifications. Insome embodiments, the notifications are utilized in the provisioningworkflow. For example, the notification may be used to monitor theprovisioning workflow. In some embodiments, the notifications are usedto manage a dependency ordering of the provisioning workflow. Forexample, post-agent provisioning jobs are run only after a completenotification is received by the provisioning server to indicate that theconfiguration agent has completed its running of its provisioning jobs.By utilizing the feedback to initiate post-agent provisioning jobs,critical dependencies in the order of provisioning jobs can bemaintained.

FIG. 1 is a block diagram illustrating an embodiment of a networkinfrastructure for automating network device provisioning. In theexample shown, network infrastructure 100 includes network device 101for provisioning, management network 103, DHCP server 105, provisioningserver 107, provisioning data store 109, asset management server 111,and asset data store 113. The elements shown may only represent a subsetof the elements of network infrastructure 100. For example, additionalelements of network infrastructure 100 may include additional routers,switches, servers, networks including optical networks, firewalls, etc.that are not shown. In various embodiments, network device 101 is arouter or a switch prepared for provisioning and physically installed onmanagement network 103. DHCP server 105 is a DHCP server for respondingto DHCP requests from network device 101. In some embodiments, DHCPserver 105 is configured to receive and respond to customized DHCPrequests with a download link to a provisioning agent. Provisioningserver 107 is one or more provisioning servers that implementprovisioning functionalities described herein. Similarly, provisioningdata store 109 is one or more data stores for storing and/or retrievingdata for provisioning. For example, provisioning data store 109 may beimplemented using one or more databases, such as MySQL databases, forstoring different provisioning configurations, provisioning loggingmessages, provisioning status of devices for remote monitoring,different versions of configuration jobs, configuration templates, etc.Asset management server 111 is one or more asset management servers thatimplement asset management functionalities such as inventory control ofassets and maintaining asset attributes. Asset data store 113 is one ormore data stores for storing and/or retrieving asset related data suchas asset attributes for provisioning. For example, asset data store 113may be implemented using one or more databases, such as MySQL databases,for storing asset attributes such as an asset identifier, serial number,make, model, and/or hostname, among other possible attributes for eachasset, such as each router, switch, server, etc. owned by a business. Insome embodiments, asset management server 111 can be queried for assetattributes stored in asset data store 113 using a hostname as the key.In the example of FIG. 1 , provisioning server 107 is communicativelyconnected to provisioning data store 109 and asset management server 111is communicatively connected to asset data store 113. In someembodiments, provisioning data store 109 and/or asset data store 113 isdirectly connected to a network (not shown) and is accessible fromprovisioning server 107 and asset management server 111, respectively,via the network.

In the example shown, network device 101, DHCP server 105, provisioningserver 107, and asset management server 111 are communicativelyconnected via management network 103. In some embodiments, managementnetwork 103 is used only for network management tasks and a separatenetwork is used for application traffic. For example, a separatehigh-speed optical network may be dedicated for carrying applicationtraffic directed to/from network device 101 that is distinct frommanagement network 103. In some cases, a single network is utilized andmanagement network 103 is used for both the management of the networkand application traffic.

In the example shown, network device 101 is installed as part of networkinfrastructure 100. For example, network device 101 is a switch or arouter that is installed in a rack in a data center and plugged intonetwork infrastructure 100 via one or more network interfaces (notshown). In some scenarios, network device 101 is installed in aco-located data center where the administrators have limited control onthe location (e.g., which rack, floor, etc.) at which network device 101is located and/or which port the network interface (not shown) ofnetwork device 101 is attached to. Although in some scenarios theadministrator installs network device 101 in a controlled data centerwith a known physical location and network location, the disclosedautomated provisioning techniques do not require knowing ahead of timethe installed location and port.

In some embodiments, configuration plans targeting different networkdevices are stored and updated on provisioning data store 109. Forexample, different configuration plans may be created to targetdifferent network devices based on differences in device attributes suchas different makes and models, different network or hardwareconfigurations, and/or for different targeted applicationservices/network usage, etc. The various configuration plans are storedin provisioning data store 109 for retrieval by provisioning server 107.Based on attributes of network device 101, a specific configuration planis generated and/or retrieved from provisioning data store 109 andapplied to network device 101. In some embodiments, provisioning datastore 109 also stores the corresponding tools, such as the most recentprogram executable(s) and necessary resources for individual jobs, suchas resource scripts, configuration files, etc., that make up aconfiguration plan. By storing (or providing a reference to) theconfiguration plans and corresponding tools and resources for performingjobs that make up the configuration plan on provisioning data store 109,the most up to date provisioning information is accessible from a knowncentral location. For example, the latest tools, executables, andresources for each job are accessible via provisioning server 107 andprovisioning data store 109. The use of a central repository helps toreduce conflicts between different configuration plan and job versionsand ensures that each device is provisioned with the correctconfiguration plan and each configuration plan includes the correct jobsand utilizes the correct tools and resources. In various embodiments,provisioning data store 109 also stores status information related tothe provisioning of network devices. For example, provisioning datastore 109 can be configured to store the progress of provisioning andwhat configuration plan and version will be, is, or has been applied toeach device. In various embodiments, the status of each device'sprovisioning is stored on provisioning data store 109 and can beaccessed via provisioning server 107. For example, a networkadministrator can access the provisioning status of an entire plane ofnetwork devices via a console or GUI, such as a web interface, usingprovisioning server 107. As another example, prior to completion ofprovisioning, provisioning data store 109 may be used to store a queueof jobs for a configuration plan for a particular device and thecorresponding status of each job.

In some embodiments, asset management server 111 and asset data store113 are utilized to store asset attributes corresponding to inventoryitems. For example, each network device has a unique identifier forinventory purposes. The unique identifier may be an asset number and canbe used to track the asset. For example, the entire lifespan of anasset, from its purchase request, purchase, installation, and/ordecommissioning can be tracked via an asset ID. The asset attributes mayalso include attributes related to the asset such as the make, model,capabilities, configuration, installation location, hardware attributes,software attributes, etc. Examples of attributes can include the datacenter the device is installed, the vendor identifier and modelidentifier, hostname, serial number, configuration plan, number ofattempts to install a particular configuration plan, etc. In variousembodiments, asset management server 111 and asset data store 113 areutilized for inventory assets other than just network devices. Examplesof these include certain inventory items that are not network devices(e.g., displays, power supplies, cabling, network interfaces, etc.) andmay not require provisioning but are nonetheless tracked as assets aspart of inventory using asset management server 111 and asset data store113.

FIG. 2 is a functional diagram illustrating a programmed computer systemfor automating network device provisioning. As will be apparent, othercomputer system architectures and configurations can be used to monitor,manage, and/or control the automation of network device provisioning. Insome embodiments, computer system 200 is a virtualized computer systemproviding the functionality of a physical computer system. Computersystem 200, which includes various subsystems as described below,includes at least one microprocessor subsystem (also referred to as aprocessor or a central processing unit (CPU)) 201. For example,processor 201 can be implemented by a single-chip processor or bymultiple processors. In some embodiments, processor 201 is a generalpurpose digital processor that controls the operation of the computersystem 200. Using instructions retrieved from memory 203, the processor201 controls the reception and manipulation of input data, and theoutput and display of data on output devices (e.g., display 209). Insome embodiments, processor 201 includes and/or is used to providefunctionality for the automation of network device provisioning. In someembodiments, computer system 200 is used to provide the functionality ofprovisioning server 107 of FIG. 1 . In some embodiments, processor 201performs or facilitates the processes described below with respect toFIGS. 3-6 .

Processor 201 is coupled bi-directionally with memory 203, which caninclude a first primary storage, typically a random access memory (RAM),and a second primary storage area, typically a read-only memory (ROM).As is well known in the art, primary storage can be used as a generalstorage area and as scratch-pad memory, and can also be used to storeinput data and processed data. Primary storage can also storeprogramming instructions and data, in the form of data objects and textobjects, in addition to other data and instructions for processesoperating on processor 201. Also as is well known in the art, primarystorage typically includes basic operating instructions, program code,data, and objects used by the processor 201 to perform its functions(e.g., programmed instructions). For example, memory 203 can include anysuitable computer-readable storage media, described below, depending onwhether, for example, data access needs to be bi-directional oruni-directional. For example, processor 201 can also directly and veryrapidly retrieve and store frequently needed data in a cache memory (notshown).

A removable mass storage device 207 provides additional data storagecapacity for the computer system 200, and is coupled eitherbi-directionally (read/write) or uni-directionally (read only) toprocessor 201. For example, storage 207 can also includecomputer-readable media such as flash memory, portable mass storagedevices, magnetic tape, PC-CARDS, holographic storage devices, and otherstorage devices. A fixed mass storage 205 can also, for example, provideadditional data storage capacity. Common examples of mass storage 205include flash memory, a hard disk drive, and an SSD drive. Mass storages205, 207 generally store additional programming instructions, data, andthe like that typically are not in active use by the processor 201. Massstorages 205, 207 may also be used to store user-generated content anddigital media for use by computer system 200. It will be appreciatedthat the information retained within mass storages 205 and 207 can beincorporated, if needed, in standard fashion as part of memory 203(e.g., RAM) as virtual memory.

In addition to providing processor 201 access to storage subsystems, bus210 can also be used to provide access to other subsystems and devices.As shown, these can include a display 209, a network interface 211, akeyboard input device 213, and pointing device 215, as well as anauxiliary input/output device interface, a sound card, speakers,additional pointing devices, and other subsystems as needed. Forexample, the pointing device 215 can be a mouse, stylus, track ball, ortablet, and is useful for interacting with a graphical user interface.

The network interface 211 allows processor 201 to be coupled to anothercomputer, computer network, or telecommunications network using one ormore network connections as shown. For example, through the networkinterface 211, the processor 201 can receive information (e.g., dataobjects or program instructions) from another network or outputinformation to another network in the course of performingmethod/process steps. Information, often represented as a sequence ofinstructions to be executed on a processor, can be received from andoutputted to another network. An interface card or similar device andappropriate software implemented by (e.g., executed/performed on)processor 201 can be used to connect the computer system 200 to anexternal network and transfer data according to standard protocols. Forexample, various process embodiments disclosed herein can be executed onprocessor 201, or can be performed across a network such as theInternet, intranet networks, or local area networks, in conjunction witha remote processor that shares a portion of the processing. Additionalmass storage devices (not shown) can also be connected to processor 201through network interface 211.

An auxiliary I/O device interface (not shown) can be used in conjunctionwith computer system 200. The auxiliary I/O device interface can includegeneral and customized interfaces that allow the processor 201 to sendand, more typically, receive data from other devices such asmicrophones, touch-sensitive displays, transducer card readers, tapereaders, voice or handwriting recognizers, biometrics readers, cameras,portable mass storage devices, and other computers.

In addition, various embodiments disclosed herein further relate tocomputer storage products with a computer readable medium that includesprogram code for performing various computer-implemented operations. Thecomputer-readable medium is any data storage device that can store datawhich can thereafter be read by a computer system. Examples ofcomputer-readable media include, but are not limited to, all the mediamentioned above and magnetic media such as hard disks, floppy disks, andmagnetic tape; optical media such as CD-ROM disks; magneto-optical mediasuch as optical disks; and specially configured hardware devices such asapplication-specific integrated circuits (ASICs), programmable logicdevices (PLDs), and ROM and RAM devices. Examples of program codeinclude both machine code, as produced, for example, by a compiler, orfiles containing higher level code (e.g., script) that can be executedusing an interpreter.

The computer system shown in FIG. 2 is but an example of a computersystem suitable for use with the various embodiments disclosed herein.Other computer systems suitable for such use can include additional orfewer subsystems. In addition, bus 210 is illustrative of anyinterconnection scheme serving to link the subsystems. Other computerarchitectures having different configurations of subsystems can also beutilized.

FIG. 3 is a flow diagram illustrating an embodiment of a process forautomating network device provisioning. In various embodiments, theprocess of FIG. 3 is used to provision and refresh the provisioning of anetwork device such as network device 101 of FIG. 1 using provisioningserver 107, DHCP server 105, and asset management server 111 of FIG. 1 .The process can be performed to provision a single device or a group ofdevices. When refreshing a network device, the process can be applied toupdate the configuration of the device using the original or a newconfiguration. Moreover, refreshes can be performed on demand,scheduled, or routinely and for a variety of reasons including whentools are updated, when the configuration requirements change, or when adevice should be reverted to its original provisioning state, as a fewexamples.

At 301, configuration targets are updated. For example, one or moreconfiguration plans are created to target different provisioningscenarios. In some scenarios, a particular configuration plan may beutilized based on a device's attributes, such as the vendor and make ofthe device and the data center the device is installed in. Differentconfiguration targets can be created and each target has a correspondingconfiguration plan or template that can include multiple jobs, includingjobs intended to be run in parallel. In various embodiments, aconfiguration plan includes jobs run by an agent on a network device aswell as jobs run prior to and after agent jobs that may not be run fromthe device. Based on a device's attributes, a different configurationplan is used. In some embodiments, the configuration plan for a deviceis based on the device's attributes, such as the device's asset ID,vendor number, model number, hostname, etc. Devices are targeted to aparticular plan based on how well the attributes of the device matchpredefined target filters. Each target can have a set of matchparameters that are compared to a device's attributes. The matchparameters may include string matching, wild card matching, regularexpression matching, etc., to target matching on one or more deviceattributes. In some embodiments, more than one configuration target maymatch a device and the closest match is considered the best match andused for selecting the proper plan. Other techniques for matching adevice's attributes to configuration targets may be utilized as well. Insome embodiments, a configuration template is used to generate theconfiguration plan. Once a target is selected, the correspondingconfiguration template allows a configuration plan to be generateddynamically and may allow for customized parameters such as hostname, IPaddress, subnet, etc. At 301, the configuration targets including theconfiguration plans/templates, corresponding jobs and resources, etc.are updated in anticipation of network device provisioning.

At 303, asset attributes for a network device are updated. For example,an asset server such as asset management server 111 is updated toinclude relevant asset attributes for a network device to beprovisioned. The asset attributes can include the asset's serial model,make, model, hostname, etc. In some embodiments, the attributes arestored in a data store such as a database by an asset management server.Using a device's asset attributes, a target configuration plan forprovisioning can be retrieved and applied to the particular device. Invarious embodiments, the device's asset attributes are maintained in theasset management server and can be later queried by requesting anasset's attributes using a key to the asset such as the hostname of thedevice.

At 305, the network device for provisioning is physically installed. Forexample, the device is installed in a particular rack on a particularfloor of a data center and one or more network interfaces of the deviceare connected. In various embodiments, an installed device can beprovisioned multiple times and different configurations can be appliedto the same device. In some scenarios the device is installed in aco-located data center by a third-party and the exact installationspecifics (e.g., floor, rack, connected port(s), etc.) are not knownahead of time. The disclosed techniques allow for automated provisioningeven without knowing these specifics ahead of time.

At 307, device provisioning is initiated. In some embodiments, a singlecommand is initiated to kick off device provisioning. For example, aconsole command specifying the hostname of the device will queue up theprovisioning of a device. The proper configuration plan will bedetermined and the device will be provisioned based on the plan. In someembodiments, more than one device can be provisioned by a singlecommand. For example, a group of devices can be specified and each willbe provisioned according to its corresponding correct configurationplan. In some embodiments, the provisioning can be initiated via a GUIby selecting an icon representing the device or specifying a hostnamefor a device name.

At 309, device provisioning is monitored. Once provisioning is initiatedat 307, provisioning is queued up and the progress can be monitored. Invarious embodiments, feedback during provisioning is reported back tothe provisioning server and the progress can be monitored by using thefeedback reported. For example, for a particular device, the particularsteps for provisioning can be queried to determine if a particular jobis queued, in progress, completed, failed, being retried, etc. Fewer oradditional monitoring states are available as appropriate. In variousembodiments, a provisioning server such as provisioning server 107 ofFIG. 1 is queried to monitor the status of provisioning. For example,once provisioning is complete, the monitoring can be used to confirmthat provisioning is complete. In some embodiments, details ofmonitoring that can be viewed and/or retrieved include the time ofcompletion, error and/or status logs from provisioning, and/or theduration of provisioning, among others. In some embodiments, a failedconfiguration job is retried and the failures and retries can bemonitored remotely.

At 311, a refresh of the device configuration is initiated, if needed.For example, a provisioned device needs to be refreshed with a newconfiguration such as a new build of one or more software packages, anew network configuration, or other updates. In various embodiments, therefresh may be initiated by issuing a console command or via a GUIsimilar to the initial initiation of provisioning at 307. In someembodiments, refreshes are initiated automatically and may be performedat regular intervals, such as part of a weekly fresh rebuild for a groupof devices. A refresh is similar to a new provisioning in that assetattributes of the device are queried and used to determine the correctconfiguration plan. In some situations, the configuration plan will bean updated plan and/or jobs in the configuration plan will be updated.By using the most up to date configuration plan, the device(s) arerefreshed with the most up to date configuration. Once a deviceconfiguration refresh is initiated, the provisioning for the refresh isqueued up and progress continues to 309 where the device provisioningcan be monitored. In this manner, once a device configuration isrefreshed, at 309, an administrator or software monitoring tool canmonitor the status of the refresh. In some scenarios, the refreshperformed at 311 is used to unconfigure a device. For example, a devicecan be wiped and prepared for removal from its installed networkenvironment.

FIG. 4 is a flow diagram illustrating an embodiment of a process for aprovisioning server for automating network device provisioning. In someembodiments, the process of FIG. 4 is performed by a provisioning serversuch as provisioning server 107 of FIG. 1 to provision a network devicesuch as network device 101 of FIG. 1 . The process may be performed forthe initial provisioning of a device as well as for a refresh of theconfiguration of a device once a device is installed in a networkenvironment. In some embodiments, the process of FIG. 4 is performed at307, 309, and/or 311 of FIG. 3 .

At 401, a network device provisioning request is received. For example,a request to initiate device provisioning is received via a consolecommand, a GUI, a shell script, a cron job, etc. In some embodiments,the request specifies a network device, for example, by specifying thedevice's hostname. In some embodiments, a group of devices may bespecified. In the example shown, a single device is specified but theapproach can be applied to a group of devices as well.

At 403, asset attributes for the device specified at 401 are retrieved.For example, asset attributes are retrieved from an asset managementserver such as asset management server 111 of FIG. 1 . In variousembodiments, a hostname or another key is used to retrieve assetattributes. Asset attributes may include the serial number, assetnumber, make, model, vendor, hardware configuration, physicalinstallation attributes such as location or data center, etc. Forexample, a particular network device may have the asset attributes thatinclude: serial number “ABCD1234,” make “WELLFLEET,” model “BNX,” andlocation “DEN,” to specify a serial number, make, model, andinstallation location.

At 405, the target configuration and associated jobs are identified.Using the asset attributes retrieved at 403, a configuration plan isidentified. For example, the asset attributes are matched to one or moreconfiguration targets and the best match is used to determine aconfiguration plan. In some embodiments, each asset attribute can beused as a field to determine a match. String matches, regularexpressions, wildcards, etc. may be used to specify a pattern thatmatches a device to a configuration target. The best and closest matchis used to select a configuration plan. For example, a wildcard willmatch all devices but a configuration target set to only match to alocation “DEN” will override the wildcard match for any devices with alocation attribute “DEN.” Using the best match out of multiple matchesallows the provisioning workflow to accommodate custom and/or uniqueprovisioning scenarios.

In some embodiments, the configuration plan is automatically generated.For example, using configuration templates associated with configurationtargets, a configuration plan is generated based on the template of thebest match to the specified device's attributes. A configuration plancan be generated using pre-defined configuration steps from a templatebut customized for the device and its attributes. For example, adevice's location and/or hardware configuration can be used to flush outconfiguration details for the configuration plan.

In some embodiments, a configuration development framework is used tomore efficiently create and run configuration jobs. Developers thatcreate and update tools that implement provisioning jobs can write theprovisioning tools by leveraging the configuration developmentframework. For example, the configuration development framework maysupport and implement common provisioning functionality such asvalidating installation steps, detecting failures, performing retries ona detected failure, logging the status of steps, parallelizing steps,etc. By building the provisioning tools using the configurationdevelopment framework, the tools used for provisioning are more stable,quicker to develop, and easier to maintain, among other benefits.

At 407, pre-agent provisioning jobs are performed. One or more jobs ofthe configuration plan are performed. In some situations, the jobs areperformed in parallel. During the performance of the jobs, feedback maybe sent to the provisioning server so that the process can be monitored.Examples of pre-agent provisioning jobs include draining traffic,suppressing alarms, erasing or wiping prior installations, testingnetwork connectivity, setting up BGP peers, etc. Only once all pre-agentprovisioning jobs are complete does processing continue to 409.

At 409, a provisioning agent is provided to the network device. Forexample, an agent for performing provisioning tasks from the device isprovided by a provisioning server. In some embodiments, a link to anagent is provided and the device will download and execute theprovisioning agent. In various embodiments, the agent may be aconfiguration agent script for executing provisioning jobs from thenetwork device.

In some embodiments, the provisioning agent is provided in response to aDHCP request from the network device to be provisioned. In response tothe DHCP request, a DHCP server, such as DHCP server 105 of FIG. 1 ,responds with a reference to a provisioning agent. The network device isable to download the provisioning agent using the provided reference,which may be a link such as a URL. In some embodiments, the agent ishosted and provided by a provisioning server, such as provisioningserver 107 of FIG. 1 . In some embodiments, the DHCP request andresponse conforms to the Zero Touch Provisioning (ZTP) frameworksupported by device hardware manufacturers. Once the network device hasreceived the provisioning agent, the provisioning agent is executed andis able to perform provisioning jobs from the local network device.During agent provisioning jobs, feedback from the network device can besent via the provisioning agent to the provisioning server. For example,feedback including the start, progress, and the completion ofprovisioning can be sent from the provisioning agent to a provisioningserver.

At 411, processing waits for agent provisioning jobs to be completed. Invarious embodiments, certain provisioning jobs are dependent on thecompletion of the provisioning jobs that must be performed by theprovisioning agent at 409. Since these jobs can only be run after thenetwork device has performed and completed its provisioning jobs, at411, the provisioning server waits for the agent to complete itsprovisioning jobs. While waiting, the progress of the agent provisioningjobs can be monitored using feedback received from the agent.

At 413, post-agent provisioning jobs are performed. In some embodiments,once agent provisioning jobs are completed, the network device isremotely accessible, for example, by remotely logging into the device.At 413, post-agent provisioning jobs are performed including jobs runfrom the network device. Jobs that run without the network device may beperformed as well. For example, a notification can be sent to informadministrators that the network device has been provisioned. Otherpost-agent provisioning jobs include undraining traffic, validatingnetwork connectivity/protocols, assigning access permissions, updatingadministrator permissions, etc. Typically, once post-agent provisioningjobs are complete, the device is now provisioned.

FIG. 5 is a flow diagram illustrating an embodiment of a process forautomating network device provisioning. The process of FIG. 5 may beused by an automated provisioning framework to perform provisioning jobsby an agent running on a network device. In some embodiments, theprocess is run after pre-agent provisioning jobs are completed, such aspart of step 409 and/or 411 of FIG. 4 . The process utilizes at least aDHCP server and a provisioning server, such as DHCP server 105 andprovisioning server 107, respectively, of FIG. 1 . In variousembodiments, the process of FIG. 5 conforms to Zero Touch Provisioning(ZTP) requirements and is consistent with network devices that supportZTP functionality that involves initiating DHCP requests to download andrun an executable script. For example, using the process of FIG. 5 , anetwork device initiates a ZTP-compatible DHCP request to at least inpart identify a downloadable script to execute. In place of thedownloadable script, a provisioning agent is provided to performprovisioning jobs and to send feedback on the process of the jobs to aprovisioning server.

At 501, a DHCP request is received. In some embodiments, the DHCPrequest is sent from a network device ready for provisioning and isreceived by a DHCP server. The DHCP request utilizes a format consistentwith Zero Touch Provisioning (ZTP) requirements. For example, the DHCPrequest is a DHCPDISCOVER message and includes particular DHCP optionsto identify the network device as a device for provisioning. Forexample, in some embodiments, an OPTION field with value code 60 isused. The DHCP message also includes a VENDOR-CLASS or similar field toidentify the hardware (e.g., vendor and model) of the network device. Insome embodiments, the DHCP request is received via a management networksuch as management network 103 of FIG. 1 . The DHCP request may be anautomated DHCP request initiated by a network device compatible with ZTPrequirements.

At 503, a DHCP request is responded to by providing provisioning agentinformation. For example, in response to a DHCPDISCOVER message receivedat 501, a DHCPOFFER message identifying a provisioning agent is providedas a DHCP response. The DHCPOFFER message includes a field, such as anOPTION field with value code 67, to identify the message as oneincluding an executable script. In some embodiments, the fieldBOOTFILE-NAME is used to identify the executable script and its valueprovides the location for retrieving a provisioning agent. In somescenarios, the location is a network path to a provisioning agent hostedby a provisioning server. In some embodiments, the field values are usedto construct an agent customized to the network device. For example,based on the model and vendor information of the network device, aparticular provisioning agent for that particular model/vendor type isprovided.

At 505, a download request for a provisioning agent is received. Forexample, an HTTP-GET request is received by a provisioning server fromthe network device to initiate the download of a provisioning agentidentified by the DHCP response of step 503. In some embodiments, thedownload request includes the full path to the provisioning agent asprovided by the value associated with the BOOTFILE-NAME field. Fornetwork devices running Zero Touch Provisioning (ZTP), the downloadrequest takes the form of a download request for a configuration script.

At 507, a determination is made on whether pre-agent provisioning hasbeen completed. In some embodiments, the determination is based on exitcodes and/or status messages from pre-agent provisioning jobs. In theevent pre-agent provisioning is complete, processing continues to 511where agent provisioning can be started. In the event pre-agentprovisioning is not complete, for example, pre-agent provisioning isongoing, processing continues to 509.

At 509, agent provisioning is delayed. Agent provisioning is delayed byresponding to the request at 505 with a response indicating that theprovisioning agent is not available. In some embodiments, an HTTP errorresponse code is used that when returned in response to the agentdownload request at 505 delays agent provisioning. The response code mayindicate an error in the download request received at 505, a servererror responding to the download request, or another appropriate errorthat delays agent provisioning. As one example, an HTTP 404 responsecode is provided to the network device requesting to download the agent.Network devices that implement Zero Touch Provisioning (ZTP) willrespond to an HTTP 404 response code by repeating the request todownload the agent. Alternative response codes may be used as well. Forexample, an HTTP 400 or an HTTP 500 response code can be used to delayagent provisioning and result in repeating the agent download request. Aresponse to the request at 505 may indicate that the request is bad orthat there is a server error. Some devices will wait a retry period,such as 15-60 seconds, before repeating the request. Processingcontinues to 505 where the provisioning server receives the retryrequest to download the provisioning agent.

In some embodiments, a substitute agent that delays provisioning isprovided in place of the actual agent that performs provisioning. Forexample, the substitute agent delays provisioning by not executing theconfiguration plan and associated jobs. Instead, the substitute agentexits with a failure code to delay provisioning. Network devices thatimplement Zero Touch Provisioning (ZTP) will respond to an unsuccessfulexit code by repeating the request to download the agent. Some deviceswill wait a retry period, such as 15-60 seconds, before repeating therequest. Processing continues to 505 where the provisioning serverreceives the retry request to download the provisioning agent.

At 511, a provisioning agent is provided for performing agentprovisioning jobs. In some embodiments, the provisioning agent isimplemented using a shell script to run on the network device and isprovided by the provisioning server in response to the request receivedat 505. For example, the configuration agent script can be provided asbinary data encoded in an HTTP reply in response to an HTTP-GET request.

At 513, agent feedback is received. Once the agent is successfullyprovided to the network device and the network device successfullydownloads the agent, the network device executes the provisioning agent.As the agent executes, feedback is sent from the agent and received bythe provisioning server. In some embodiments, the provisioning serverwill log and/or display the agent feedback. The feedback may includemessages indicating that the agent is starting, a particular job of theconfiguration is starting, a job is complete, agent provisioning iscomplete, etc. Different types of feedback from the agent may bereceived by the provisioning server. In some embodiments, the feedbackis sent by encoding the feedback using an HTTP-GET request. For example,the feedback can be encoded in the URL requested. As an example, the URLmay include the string “/start/” appended with the serial number of thenetwork device to indicate which particular network device is startingits provisioning. As another example, the URL may include the string“/complete/” appended with the unique identifier. The unique identifiermay be a unique identifier provided as a result of a start call. Theunique identifier may be used to identify the job that is beingcompleted and/or which particular network device has completedprovisioning. Other parameters and messages can be similarly encoded inthe URL. Depending on the message, a different response is provided bythe provisioning server. For example, a simple acknowledgement may beprovided in response to a complete message. As another example, aparticular job and job identifier can be provided to indicate theupcoming job to run. In some embodiments, the feedback is sent via analternative method such as encoding the feedback using an HTTP-POSTrequest.

At 515, the completion of agent provisioning jobs is acknowledged. Forexample, in response to an HTTP request with feedback indicating thatprovisioning is complete, the provisioning server responds with anacknowledgement message to confirm receipt of the completion message. Insome embodiments, the acknowledgement message is optional.

FIG. 6 is a flow diagram illustrating an embodiment of a process forreceiving feedback from a provisioning agent for a network device. Insome embodiments, the feedback is received by a provisioning server froma provisioning agent. The feedback may be logged to a data store such asa provisioning MySQL database by the provisioning server. Further, usersand tools can monitor the progress of agent provisioning by reviewingthe feedback. For example, the feedback can include progress updates,job completion, errors encountered, and retries performed, among otherstatus updates or notifications. In some embodiments, the process ofFIG. 6 is performed by a provisioning server such as provisioning server107 of FIG. 1 on a network device such as network device 101 of FIG. 1 .In some embodiments, the process of FIG. 6 is performed at 513 and/or515 of FIG. 5 once a provisioning agent has been installed onto thenetwork device for provisioning. In various embodiments, the feedbackprovided by the provisioning agent conforms to the Zero TouchProvisioning (ZTP) framework that allows a configuration script to bedownloaded by initiating an automated DHCP request.

At 601, agent start feedback is received. For example, a feedbackmessage corresponding to the start of provisioning initiated by aprovisioning agent is received. In some embodiments, the feedback issent by encoding the feedback using an HTTP-GET request. For example,the feedback can be encoded in the URL requested. As an example of oneencoding scheme, the URL may include the string “/start/” appended withthe serial number of the network device to indicate which particularnetwork device is starting provisioning. Once the request for thisparticular URL is received by the provisioning server, the URL isdecoded into a start feedback message for the network device. Othertechniques for encoding the feedback message can be used as well. Forexample, the message can be encoded using an HTTP-POST request to theprovisioning server. Additional arguments or parameters can be encodedin the URL as well.

At 603, the start of agent provisioning is acknowledged. For example, inresponse to the agent start feedback message received at 601, anacknowledgement is sent from the provisioning server to the networkdevice. In some embodiments, the acknowledgement includes a job numbercorresponding to a job identifier field. In some embodiments, the jobnumber corresponds to a particular job to be executed by the agent onthe network device as part of a configuration plan.

At 605, agent progress feedback is received. For example, the agent maysend a feedback message indicating key events related to provisioning,such as the start or completion of a job that is part of a configurationplan. The progress feedback may also indicate the completion progress,any errors encountered, status updates, an attempt to retry, the numberof retries before a successful outcome, or other progress feedback asappropriate. Progress feedback messages may be used to monitor and/orsolve provisioning issues. In some embodiments, the feedback messagesare received as messages encoded in an HTTP-GET or HTTP-POST request. Invarious embodiments, the progress feedback other than notifying whenprovisioning is complete is optional and may not be provided by aprovisioning agent or may only be provided during some jobs performed bythe provisioning agent.

In some embodiments, the progress feedback is a complete message thatindicates agent provisioning has been completed, that is, allprovisioning jobs initiated by the agent have been run and successfullycompleted. For example, a completion message is received in an HTTP-GETmessage. As one technique for encoding the completion message, the URLrequested includes a “complete” string to indicate the provisioningperformed by the agent has completed. The URL may also include a uniqueidentifier corresponding to the provisioning and/or the network device.Other encoding techniques are possible as well. In some embodiments, thejob number that is completed is included in the encoding along with anexit code.

At 607, a determination is made whether the progress feedback receivedat 605 is an agent complete message indicating that agent provisioningis complete. In response to an agent complete message, processingcontinues to 611. In response to a progress feedback that is not anagent complete message, processing continues to 609.

At 609, agent progress feedback received at 605 is acknowledged. In someembodiments, different acknowledgements are used depending on the typeof feedback message. For example, progress feedback corresponding to jobcompletion may include the information related to the next job(s) to beexecuted as part of the configuration plan. For example, multiple jobscan be specified to run multiple jobs in parallel and each job isspecified in the acknowledgement. In some scenarios, a simpleacknowledgement with no additional information is utilized. For example,progress feedback corresponding to the start of a job may beacknowledged with a simple “OK” confirming the receipt of the progressfeedback message at 605. In some situations, an acknowledgement isoptional and may not be required by the agent to continue provisioning.In all cases, progress continues to 605 where additional feedback isreceived.

At 611, agent completion is acknowledged. The provisioning serveracknowledges agent completion in response to receiving the agentcomplete feedback at 605. In various embodiments, once agentprovisioning is complete, the network device is now accessible remotely,for example, by connecting to an ssh server running on the networkdevice. Additional provisioning steps can be completed without the needof a provisioning agent running on the device. For example, theprovisioning server can now remotely and securely connect to the networkdevice to run post-agent provisioning jobs from the device withoutrequiring the provisioning agent. In some embodiments, the post-agentprovisioning jobs are performed at 413 of FIG. 4 .

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, the invention is not limitedto the details provided. There are many alternative ways of implementingthe invention. The disclosed embodiments are illustrative and notrestrictive.

What is claimed is:
 1. A method, comprising: receiving a deviceprovisioning configuration request from a network device to beconfigured, wherein the network device provided the request to a networklocation obtained via a Dynamic Host Configuration Protocol request madeautomatically by the network device; dynamically and automaticallygenerating a customized configuration jobs plan at least in part on avendor identifier and a model identifier of the network device to beconfigured and a target network usage in a data center among differentnetwork usage options, wherein generating the customized configurationjobs plan includes selecting a configuration jobs plan specifyingconfiguration jobs among a plurality of different stored configurationplans specifying different sets of agent executed configuration jobs andthe selected configuration jobs plan is identified based on at least apartial pattern match to the vendor identifier or the model identifierof the network device, the configuration jobs include a plurality ofdifferent jobs local to the network device and wherein the plurality ofdifferent jobs specified by the customized configuration jobs plan basedat least in part on the vendor identifier and the model identifierincludes a job of locally installing software and a different job ofsetting up local firewall rules; providing a configuration agent scriptin response to the device provisioning configuration request; receivingnotifications associated with a provisioning progress from the networkdevice via the configuration agent script, wherein the notificationsprovide different status updates for each of the plurality of differentjobs specified by the automatically generated configuration jobs plancustomized for the network device based at least in part on the vendoridentifier and the model identifier; and utilizing at least one of thenotifications in a provisioning workflow.
 2. The method of claim 1,wherein the at least one of the notifications in the provisioningworkflow is utilized to manage a dependency ordering of the provisioningworkflow.
 3. The method of claim 1, wherein the notifications include astart notification.
 4. The method of claim 3, further comprising, inresponse to the start notification, providing a notificationacknowledgement, wherein the notification acknowledgement includes a jobidentifier indicating a provisioning job to perform by the configurationagent script.
 5. The method of claim 1, wherein the notificationsinclude a notification indicating a completion.
 6. The method of claim1, wherein at least one of the notifications is encoded as an HTTPrequest.
 7. The method of claim 6, wherein the HTTP request includes aunique identifier associated with the network device to be configured.8. The method of claim 1, wherein the Dynamic Host ConfigurationProtocol request includes at least the vendor identifier and the modelidentifier of the network device to be configured.
 9. The method ofclaim 1, wherein the Dynamic Host Configuration Protocol request isinitiated as part of a Zero Touch Provisioning process.
 10. The methodof claim 1, wherein the network device to be configured is included in agroup of network devices to be configured, wherein the group of networkdevices is configured in response to an initiation of a singleprovisioning command.
 11. The method of claim 1, wherein utilizing theat least one of the notifications in the provisioning workflow includesdelaying a post-configuration agent script provisioning job until acomplete notification is received.
 12. The method of claim 1, furthercomprising: performing pre-configuration agent script provisioning jobs.13. The method of claim 12, wherein the providing of the configurationagent script in response to the device provisioning configurationrequest is delayed at least until the pre-configuration agent scriptprovisioning jobs are complete.
 14. The method of claim 1, wherein: theconfiguration jobs plan was generated using attributes of the networkdevice to be configured, wherein the attributes are retrieved from anasset server.
 15. The method of claim 14, wherein the configuration jobsplan generated using the attributes of the network device to beconfigured is generated using the selected configuration plan.
 16. Themethod of claim 14, wherein the attributes include at least a serialnumber, a make information, a model information, or a locationinformation.
 17. The method of claim 16, wherein the selectedconfiguration jobs plan is determined by performing string comparisonsand wildcard matches using a plurality of configuration templates. 18.The method of claim 1, wherein the configuration agent script isconfigured to execute one or more provisioning jobs, wherein a recentversion of a tool corresponding to at least one of the one or moreprovisioning jobs is retrieved via a provisioning database.
 19. Asystem, comprising: a processor; and a memory coupled with theprocessor, wherein the memory is configured to provide the processorwith instructions which when executed cause the processor to: receive adevice provisioning configuration request from a network device to beconfigured, wherein the network device provided the request to a networklocation obtained via a Dynamic Host Configuration Protocol request madeautomatically by the network device; dynamically and automaticallygenerate a customized configuration jobs plan based at least in part ona vendor identifier and a model identifier of the network device to beconfigured and a target network usage in a data center among differentnetwork usage options, wherein being configured to generate thecustomized configuration jobs plan includes being configured to select aconfiguration jobs plan specifying configuration jobs among a pluralityof different stored configuration plans specifying different sets ofagent executed configuration jobs and the selected configuration jobsplan is identified based on at least a partial pattern match to thevendor identifier or the model identifier of the network device, theconfiguration jobs include a plurality of different jobs local to thenetwork device and wherein the plurality of different jobs specified bythe customized configuration jobs plan based at least in part on thevendor identifier and the model identifier includes a job of locallyinstalling software and a different job of setting up local firewallrules; provide a configuration agent script in response to the deviceprovisioning configuration request; receive notifications associatedwith a provisioning progress from the network device via theconfiguration agent script, wherein the notifications provide differentstatus updates for each of the plurality of different jobs specified bythe automatically generated configuration jobs plan customized for thenetwork device based at least in part on the vendor identifier and themodel identifier; and utilize at least one of the notifications in aprovisioning workflow.
 20. A computer program product, the computerprogram product being embodied in a non-transitory computer readablestorage medium and comprising computer instructions for: receiving adevice provisioning configuration request from a network device to beconfigured, wherein the network device provided the request to a networklocation obtained via a Dynamic Host Configuration Protocol request madeautomatically by the network device; dynamically and automaticallygenerating a customized configuration jobs plan based at least in parton a vendor identifier and a model identifier of the network device tobe configured and a target network usage in a data center amongdifferent network usage options, wherein generating the customizedconfiguration jobs plan includes selecting a configuration jobs planspecifying configuration jobs among a plurality of differentconfiguration plans specifying different sets of agent executedconfiguration jobs and the selected configuration jobs plan isidentified based on at least a partial pattern match to the vendoridentifier or the model identifier of the network device, theconfiguration jobs include a plurality of different jobs local to thenetwork device and wherein the plurality of different jobs specified bythe customized configuration jobs plan based at least in part on thevendor identifier and the model identifier includes a job of locallyinstalling software and a different job of setting up local firewallrules; providing a configuration agent script in response to the deviceprovisioning configuration request; receiving notifications associatedwith a provisioning progress from the network device via theconfiguration agent script, wherein the notifications provide differentstatus updates for each of the plurality of different jobs specified bythe automatically generated configuration jobs plan customized for thenetwork device based at least in part on the vendor identifier and themodel identifier; and utilizing at least one of the notifications in aprovisioning workflow.